Monitoring apparatus, monitoring system, and monitoring method

ABSTRACT

A monitoring apparatus according to embodiments of the invention generates and outputs an output moving image obtained by changing an image area of a person detected from an imaged moving image of a monitoring area to a mask image. The monitoring apparatus includes a person image analyzer including a person detector which detects a person from the imaged moving image and acquires area information regarding each person and an attribute determinator which determines attributes for each person detected from the imaged moving image, a mask image setter which sets the mask image by the attribute, and a moving image output controller which generates and outputs the output moving image obtained by changing the image area for each person to the mask image by the attribute, based on the area information and the attribute for each person output from the person image analyzer.

TECHNICAL FIELD

The present invention relates to a monitoring apparatus, a monitoringsystem, and a monitoring method that generate and output an outputmoving image obtained by changing an image area of a person detectedfrom an imaged moving image of a monitoring area to a mask image.

BACKGROUND OF THE INVENTION

A monitoring system of installing a camera which images the inside ofthe store and monitoring a person in the store through a moving image ofthe camera is widely used in a store such as a convenience store, but ina case of using the imaged moving image for a purpose other than themonitoring, that is, for a purpose of marketing analysis or serviceimprovement, it is necessary to protect privacy of customers.

Meanwhile, Japanese Patent Unexamined Publication No. 2013-186838discloses a technology of displaying feature points on an image obtainedby changing the entire person area in each image of a moving imagecaptured by a camera to an image covered with a predetermined color, andgrasping how the person moves. Japanese Patent No. 5159381 discloses atechnology of generating a background image having no foreground imageof a person, performing mask processing or filter processing withpartial or entire translucency to foreground image, and generating animage where a face or a figure of a person is hardly specified.

In order to grasp the action of the person from the moving image, towatching the person while distinguishing the person from the backgroundis unavoidable, but as the technologies in the related art, the imagearea of the person is replaced with the specific image to easilydiscriminate between the background and the person, and accordingly, itis easy to grasp the action of the person.

Meanwhile, it is desired to grasp what kind of tendency the action ofthe person has by attribute of a person (gender, age, and the like), butin the technologies in the related art, there are no concerns about sucha demand, and it is difficult to efficiently perform an operation ofgrasping the action tendency of the person by attribute of a person.Particularly, in a case where a lot of persons are shown in a movingimage, it is necessary to watch the actions of the persons whiledifferentiating each person and determining customer base of eachperson, and accordingly, a burden of a user is large when the playbacktime of the moving image is long, and a technology of efficientlyperforming the operation of grasping the action tendency of the personby attribute of a person is desired.

It is desired to grasp what kind of tendency the action of the personhas by date and time (day or time zone), but in the technologies in therelated art, there is no concerns about such a demand, and it isdifficult to efficiently perform an operation of grasping the actiontendency of the person by date and time. Particularly, in order to graspthe action tendency of the person by date and time, it is necessary towatch the action of the person while comparing the moving images havingdifferent date and time from each other, and accordingly, a burden of auser is large when the playback time of the moving image is long, and atechnology of efficiently performing the operation of grasping theaction tendency of the person by date and time is desired.

It is desired to grasp how products change with the action of theperson, but in the technologies in the related art, there is no concernsabout such a demand, and it is difficult to efficiently performing anoperation of grasping a changing situation of the product. Particularly,it is difficult to grasp the state of the products, in a moving imagewhere the action of the person can be grasped, and accordingly, a burdenof a user is large when the playback time of the moving image is long,and a technology of efficiently perform the operation of grasping thechanging situation of the product with the action of the person isdesired.

SUMMARY OF THE INVENTION

A monitoring apparatus according to an embodiment of the inventiongenerates and outputs an output moving image obtained by changing animage area of a person detected from an imaged moving image of amonitoring area to a mask image. The monitoring apparatus includes aperson image analyzer including a person detector which detects a personfrom the imaged moving image and acquires area information regardingeach person and an attribute determinator which determines attributesfor each person detected from the imaged moving image, a mask imagesetter which sets the mask image by the attribute, and a moving imageoutput controller which generates and outputs the output moving imageobtained by changing the image area for each person to the mask image bythe attribute, based on the area information and the attribute for eachperson output from the person image analyzer.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is an overall configuration diagram of a monitoring systemaccording to a first exemplary embodiment;

FIG. 2 is a plan view of a store illustrating a layout of the store andan installation situation of cameras 1;

FIG. 3 is a functional block diagram showing a schematic configurationof PC 3;

FIG. 4 is an explanatory diagram showing a monitoring screen displayedon monitor 4;

FIG. 5 is an explanatory diagram showing a mask image setting screendisplayed on monitor 4;

FIGS. 6A and 6B are explanatory diagrams showing main parts of movingimages displayed on moving image display part 48 of a monitoring screen;

FIG. 7 is a functional block diagram showing a schematic configurationof PC 3 of a monitoring system according to a second exemplaryembodiment;

FIG. 8 is an explanatory diagram showing a monitoring screen displayedon monitor 4;

FIG. 9 is an explanatory diagram showing a mask image setting screendisplayed on monitor 4;

FIGS. 10A and 10B are explanatory diagrams showing moving images on twoimaging dates and times;

FIGS. 11A and 11B are explanatory diagrams showing main parts of movingimage display part 48 of a monitoring screen;

FIG. 12 is a functional block diagram showing a schematic configurationof PC 3 of a monitoring system according to a third exemplaryembodiment;

FIG. 13 is an explanatory diagram showing a monitoring screen displayedon monitor 4;

FIG. 14 is an explanatory diagram showing a mask image setting screendisplayed on monitor 4;

FIGS. 15A and 15B are explanatory diagrams showing main parts of movingimage display part 48 of a monitoring screen;

FIGS. 16A, 16B, and 16C are explanatory diagrams showing main parts ofmoving image display part 48 of a monitoring screen; and

FIG. 17 is an explanatory diagram showing another example of amonitoring screen displayed on monitor 4.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The invention is proposed to solve problems of technologies in therelated art, and a main object thereof is to provide a monitoringapparatus, a monitoring system, and a monitoring method which can make auser efficiently perform an operation of grasping an action tendency ofa person by attribute of a person.

The invention is proposed to solve problems of technologies in therelated art, and a main object thereof is to provide a monitoringapparatus, a monitoring system, and a monitoring method which can make auser efficiently perform an operation of grasping an action tendency ofa person by date and time.

The invention is proposed to solve problems of technologies in therelated art, and a main object thereof is to provide a monitoringapparatus, a monitoring system, and a monitoring method which can make auser efficiently perform an operation of grasping a changing situationof a product with the action of the person.

According to a first aspect of the invention, there is provided amonitoring apparatus which generates and outputs an output moving imageobtained by changing an image area of a person detected from an imagedmoving image of a monitoring area to a mask image, the apparatusincluding: a person image analyzer including a person detector whichdetects a person from the imaged moving image and acquires areainformation regarding each person and an attribute determinator whichdetermines attributes for each person detected from the imaged movingimage; a mask image setter which sets the mask image by the attribute;and a moving image output controller which generates and outputs theoutput moving image obtained by changing the image area for each personto the mask image by the attribute, based on the area information andthe attribute for each person output from the person image analyzer.

In this case, since the mask images are displayed by attributes, it ispossible to simply grasp the attribute of the person appearing on themoving image, and therefore, it is possible to efficiently perform theoperation of grasping the action tendency of the person for eachattribute of the person.

According to a second aspect of the invention, the mask image setter mayset the mask image by the attribute, according to an input operation bya user of selecting the mask image by the attribute.

In this case, since it is possible for a user to freely set the maskimage, it is possible to improve convenience to a user.

According to a third aspect of the invention, the mask image may be setto discriminate the attribute, by changing at least any of displayelements such as a color, shading, a pattern, and an outline, accordingto the attribute.

In this case, it is possible to simply discriminate the persons havingdifferent attributes, by differences of the mask images.

According to a fourth aspect of the invention, the monitoring apparatusmay further include a background image generator which generates abackground image from the imaged moving image, and the moving imageoutput controller generates the output moving image where thetranslucent mask image is superposed on the background image.

In this case, since the mask image has translucency, the backgroundimage is visible through the mask image, and therefore, it is possibleto observe the action of the person while watching the situation of themonitoring area shown on the background image.

According to a fifth aspect of the invention, there is provided amonitoring system which generates and outputs an output moving imageobtained by changing an image area of a person detected from an imagedmoving image of a monitoring area to a mask image, the system including:cameras which image the monitoring area; a plurality of informationprocessing apparatuses; a person image analyzer including a persondetector which detects a person from the imaged moving image andacquires area information regarding each person and an attributedeterminator which determines attributes for each person detected fromthe imaged moving image; a mask image setter which sets the mask imageby the attribute; and a moving image output controller which generatesand outputs the output moving image obtained by changing the image areafor each person to the mask image by the attribute, based on the areainformation and the attribute for each person output from the personimage analyzer.

In this case, in the same manner as in the first aspect, it is possiblefor a user to efficiently perform the operation of grasping the actiontendencies of the persons by the attributes of the persons.

According to a sixth aspect of the invention, there is provided amonitoring method which causes an information processing apparatus toperform a process of generating and outputting an output moving imageobtained by changing an image area of a person detected from an imagedmoving image of a monitoring area to a mask image, the method including:a step including a step of detecting a person from the imaged movingimage and acquiring area information regarding each person and a step ofdetermining attributes for each person detected from the imaged movingimage; a step of setting the mask image by the attribute; and a step ofgenerating and outputting the output moving image obtained by changingthe image area for each person to the mask image by the attribute, basedon the area information and the attribute for each person.

In this case, in the same manner as in the first aspect, it is possiblefor a user to efficiently perform the operation of grasping the actiontendencies of the persons by the attributes of the persons.

Hereinafter, the embodiments of the invention will be described withreference to the drawings.

First Exemplary Embodiment

FIG. 1 is an overall configuration diagram of a monitoring systemaccording to a first exemplary embodiment. This monitoring system isbuilt for a chain of retail stores such as convenience stores, andincludes camera (imaging apparatus) 1, recorder (image recordingapparatus) 2, PC (monitoring apparatus) 3, monitor (display apparatus)4, which are provided in a plurality of stores, and PC 11 and monitor 12which are provided in a head office which generally manages theplurality of stores.

Cameras 1 are installed in appropriate places in the store, the insideof the store is imaged by cameras 1, and image information obtained isrecorded in recorder 2. In PC 3 installed in the store or PC 11installed in the head office, it is possible to browse the image of theinside of the store imaged by cameras 1 in real time, it is possible tobrowse the image of the inside of the store in the past recorded inrecorder 2, and therefore, it is possible to check the situation in thestore, in the store or in the head office.

PC 3 installed in the store is configured as a monitoring apparatuswhich monitors the state of the person or the product in the store. Itis possible to browse monitoring information generated by PC 3 in PC 3,and it is also possible to browse the monitoring information in PC 11 bytransmitting the monitoring information to PC 11 installed in the headoffice. PC 3 and PC 11 are configured as a browsing apparatus whichbrowses monitoring information. PC 3 and PC 11 include a processor (CPU)and a program memory. The CPU of PC 3 is realized by executing programs(instructions) for monitoring. The programs may be introduced to PC 3 asan information processing apparatus in advance to configure the PC as adedicated apparatus, or may be supplied to a user by being recorded in asuitable program recording medium as an application program to beoperated on a general-use OS, or through a network.

Next, a layout of the store and an installation state of cameras 1 willbe described using a convenience store as an example. FIG. 2 is a planview of a store illustrating the layout of the store and theinstallation state of cameras 1.

In the store, an entrance, display shelves (display areas), and cashregisters are provided. The display shelves are installed to be dividedinto types of products such as a lunch box, a PET bottle, onigiri, andthe like. A display shelf for fast food is installed next to the cashregisters.

A customer enters the store from the entrance, moves in the storethrough passages between the display shelves, heads for the cashregister with a product when a desired product is found, makes a paymentat the cash register, and exits the store from the entrance.

The plurality of cameras 1 which image the inside of the store(monitoring area) are installed in the store. Particularly, in theexample shown in FIG. 2, an omnidirectional camera including an imagingrange at 360 degrees using a fisheye lens is employed as camera 1, andit is possible to image the products displayed on the display shelvesand to image the person staying in front of the display shelves or thecash registers by camera 1.

Next, the process performed by PC 3 shown in FIG. 1 will be describedwith based on a functional block diagram. FIG. 3 is a functional blockdiagram showing a schematic configuration of PC 3.

PC 3 includes a monitoring unit 31. Monitoring unit 31 causes PC 3 tofunction as a monitoring system which monitors the inside of the store.Monitoring unit 31 can control the operations of cameras 1 and recorder2, can browse the image of the inside of the store imaged by cameras 1in real time, and can browse the image of the inside of the storerecorded by recorder 2.

PC 3 includes person image analyzer 32, background image generator 33,moving image output controller 34, and mask image setter 35.

Person image analyzer 32 analyzes a moving image imaged by camera 1 andacquires information regarding the person, and includes person detector37 and attribute determinator 38.

Person detector 37 performs a process of detecting the person from theimaged moving image and acquiring area information for each person(positional information regarding an image area of a person). Attributedeterminator 38 performs a process of determining attribute (gender,age, and the like) for each person detected from the imaged movingimage. For the processes performed by person detector 37 and attributedeterminator 38, a well-known image recognition technology may be used.It is desired to perform a person tracking process of correlating persondetection results for each frame of the imaged moving image by personimage analyzer 32, and therefore, it is possible to perform attributedetermination with high accuracy.

Mask image setter 35 performs a process of setting a mask image byattribute. Particularly, in the embodiment, a process of setting a maskimage by attribute is performed according to an input operation by auser for selecting a mask image for each attribute of the person. Themask image by attribute may be set in advance.

Background image generator 33 performs a process of generating abackground image from the imaged moving image. This background image isobtained by removing the image of the person (foreground image) from theimaged moving image. Each image of the person from the plurality ofmoving images recently imaged may be removed and the background imagemay be synthesized from the plurality of images. The imaged moving imagewhen there are no persons, for example, the moving image imaged beforeopening the store, may be prepared in advance as the background image.

Moving image output controller 34 performs a process of generating andoutputting an output moving image obtained by changing the image area ofthe person to a mask image, based on the area information and attributefor each person output by person image analyzer 32. In the embodiment, aprocess of generating a mask image having an outline corresponding tothe image area of the person in the imaged moving image and superposingthe mask image on the background image generated by background imagegenerator 33 is performed.

The mask image is set for a user to discriminate the attribute bychanging at least any of display elements such as a color, shading, apattern (design), and an outline, according to the attribute.Particularly, in the embodiment, the outline of the image area of theperson is generated from the area information regarding each person(positional information regarding the image area of the person) and theinside of the outline is covered with a translucent image. The outlinemay be set not to be displayed or the outline may be displayed to bediscriminated from the covered area by changing a color or shading.

Next, a monitoring screen displayed on monitor 4 shown in FIG. 1 will bedescribed. FIG. 4 is an explanatory diagram showing the monitoringscreen displayed on monitor 4.

This monitoring screen is provided for a user to check the situation inthe store for a purpose of marketing analysis or service improvement,and the monitoring screen is provided with store selection part 41,start button 42, setting button 43, operation mode selection part 44,date and time display part 45, date and time setting button 46,explanatory note display part 47, moving image display part 48, anddisplay time point operation part 49.

A user selects the store from a pull-down menu in store selection part41. Start button 42 starts the monitoring process by the PC. Settingbutton 43 sets various conditions in the monitoring process, and in theembodiment, when setting button 43 is operated, a mask image settingscreen (see FIG. 5) is displayed as a pop-up. Operation mode selectionpart 44 selects an operation mode. Operation modes such as selling areamonitoring, product monitoring, display alert, and stockout alert areprepared, and the selling area monitoring is selected herein.

Date and time display part 45 displays date and time. Date and timesetting button 46 sets the date and time. When date and time settingbutton 46 is operated, a date and time setting screen (not shown) isdisplayed, and by selecting the date and time herein, the selected dateand time is displayed on date and time display part 45, and the movingimage on the selected date and time is displayed on moving image displaypart 48.

Explanatory note display part 47 explains the mask image displayed onmoving image display part 48. In the embodiment, the mask image set bythe attribute of the person (gender, age, and the like) is displayed onexplanatory note display part 47. In the example shown in FIG. 4, theage is set as all generations, but in a case where the mask image is setfor each generation of the teenagers to sixties, for example, the maskimage of each generation is displayed on explanatory note display part47.

The moving image (video) of the store is displayed on moving imagedisplay part 48. Particularly, in the example shown in FIG. 4, twopanoramic moving images of the moving image imaged by camera 1 which isthe omnidirectional camera, are displayed. The entire store is shown inthe two moving images, and it is possible to check the situation of theentire store. In moving image display part 48, the moving image imagedby camera 1 is displayed as it is in a normal playback mode, and themoving image obtained by changing the image area of the person to themask image is displayed in a mask playback mode. The moving imagedisplayed on moving image display part 48 will be described later indetail.

Display time point operation part 49 adjusts the time point of themoving image displayed on moving image display part 48. By operatingdisplay time point operation part 49, it is possible to replace themoving image with a moving image on the desired time point.Specifically, when slider 50 is moved using input device 6 such as amouse, the moving image at the time point indicated by slider 50 isdisplayed on moving image display part 48. Accordingly, it is possibleto check the situation of the store on the desired time point with themoving image on moving image display part 48. When playing the movingimage of moving image display part 48, it is possible to set normalplayback, fast-forward playback, or rewind playback by using operationbutton 51. In this case, even when playing the moving image recorded fora long time in the mask playback mode, it is possible to efficientlycheck the moving image.

Next, the operation of setting the mask image by attribute will bedescribed. FIG. 5 is an explanatory diagram showing a mask image settingscreen displayed on monitor 4.

The mask image setting screen is provided for a user to designate themask image for each attribute of the person, and the mask image settingscreen is provided with person attribute selection part 61, mask imageselection part 62, setting button 63, and set contents display part 64.

A user selects the attribute of the person (gender, age, and the like)from a pull-down menu in person attribute selection part 61. It ispossible to select any one of generations from teenagers to sixties andall generations, for example, in person attribute selection part 61. Theattribute of the person may be set as adults and children. A userselects from a title menu the mask image corresponding to the attributeselected in person attribute selection part 61, in mask image selectionpart 62.

When the attribute of the person and the mask image correspondingthereto are selected in person attribute selection part 61 and maskimage selection part 62, and setting button 63 is operated, the setcontents, that is, the mask images by attribute, are displayed in setcontents display part 64. In the example shown in FIG. 5, the maskimages for male and female are set by setting the age as allgenerations.

Next, the moving image displayed on moving image display part 48 of themonitoring screen shown in FIG. 4 will be described. FIGS. 6A and 6B areexplanatory diagrams showing main parts of the moving image displayed onmoving image display part 48 of the monitoring screen shown in FIG. 4.

As shown in FIG. 6A, the moving image imaged by camera 1 is displayed asit is in the normal playback mode. In contrast, as shown in FIG. 6B, themoving image obtained by changing the image area of the person to themask image is displayed in the mask playback mode after performing themask image setting. In this moving image, the image area of the personin the imaged moving image of FIG. 6A is replaced with the mask image byattributes (male and female, herein). Since the mask image hastranslucency, the background image is visible through the mask image.

When playing the imaged moving image, it is preferable to display anicon for play mode discrimination to be superposed with the movingimage, so that a user can discriminate in which playback mode the movingimage is played. It is also possible to discriminate the playback modeby displaying a color of the moving image display frame in the maskplayback mode and a color of the moving image display frame in thenormal playback mode, with different colors.

In the first exemplary embodiment as described above, since the maskimages by attribute, for example, male and female, are displayed, it ispossible to simply grasp the attribute of the person appearing on themoving image, and therefore, it is possible to efficiently perform theoperation of grasping the action tendency of the person for eachattribute of the person.

The gender and the age have been described as the examples of theattribute of the person, but various attributes such as a direction of aperson, the race, adults/children, and a type (female office worker ormale office worker) can be set as the attributes obtained by using awell-known person recognition technology. Particularly, when using theattribute of the direction of the person, it is possible to grasp ataste of each person with respect to the product in the mask playbackmode.

Second Exemplary Embodiment

Next, a second exemplary embodiment will be described. The points notparticularly stated herein are the same as in the first exemplaryembodiment.

FIG. 7 is a functional block diagram showing a schematic configurationof PC 3 of the monitoring system according to the second exemplaryembodiment. In the second exemplary embodiment, in the same manner as inthe first exemplary embodiment, PC 3 includes monitoring unit 31, personimage analyzer 32, background image generator 33, moving image outputcontroller 34, and mask image setter 35, and PC 3 in the secondexemplary embodiment further includes date and time setter 71.

In the second exemplary embodiment, in the same manner as in the firstexemplary embodiment, it is possible to display the mask image by theattribute of the person (gender, age, and the like), but it is alsopossible to display the mask image by date and time. It is also possibleto have a configuration of displaying only the mask image by date andtime, without displaying the mask image by the attribute of the person.

Date and time setter 71 performs a process of setting the plurality ofdates and times according to the input operation by a user ofdesignating the imaging date and time. Mask image setter 35 performs aprocess of setting a mask image by the imaging date and time and by theattribute of the person, according to the input operation by a user ofselecting the mask image by the imaging date and time and by theattribute of the person. Person image analyzer 32 performs a process ofacquiring the imaged moving image on the plurality of dates and timesset by date and time setter 71 and acquiring the person area informationfor each person (positional information regarding the image area of theperson) by the imaging date and time.

Moving image output controller 34 performs a process of generating andoutputting an output moving image obtained by changing the image area ofthe person by the plurality of imaging dates and times to a mask imageby the imaging date and time and by the attribute of the person, basedon the area information regarding each person output from person imageanalyzer 32, and the output images are the mask images for each personon the plurality of dates and times which are displayed so as todiscriminate the imaging dates and times and the attribute of the personon one background image.

In the second exemplary embodiment, the mask image is set for a user todiscriminate the attribute and the imaging dates and times by changingat least any of display elements such as a color, shading, a pattern,and an outline, according to the attribute and the imaging date andtime. Particularly, in the embodiment, the inside of the outline of theimage area of the person is covered with a translucent image.

Next, a monitoring screen displayed on monitor 4 will be described. FIG.8 is an explanatory diagram showing the monitoring screen displayed onmonitor 4.

Even in the second exemplary embodiment, a monitoring screen having aconfiguration approximately similar to that of the first exemplaryembodiment (see FIG. 4) is displayed on monitor 4, and in the monitoringscreen, a mask image set by the plurality of imaging dates and times andby the attribute of the person is displayed on explanatory note displaypart 47.

Next, the operation of setting the mask image by the attribute and theimaging dates and times will be described. FIG. 9 is an explanatorydiagram showing a mask image setting screen displayed on monitor 4.

The mask image setting screen is provided for a user to designate themask image for each attribute of the person and for each imaging datesand times, and the mask image setting screen is provided with personattribute selection part 81, date and time selection part 82, mask imageselection part 83, setting button 84, and setting contents display part85.

A user selects the attribute of the person (gender, age, and the like)from a pull-down menu in person attribute selection part 81. A userselects the date and time (year, month, date, and time zone) from apull-down menu in date and time selection part 82. A user selects theorder of priority from a pull-down menu in date and time selection part82. A user selects the mask image corresponding to the attributeselected in person attribute selection part 81 and the imaging date andtime selected in date and time selection part 82, from a title menu inmask image selection part 83.

When the attribute of the person, the imaging date and time, and themask image corresponding thereto are selected in person attributeselection part 81, date and time selection part 82, and mask imageselection part 83, and setting button 84 is operated, the set contents,that is, the mask images by attribute and by imaging date and time aredisplayed in set contents display part 85. This setting operation is atleast performed twice, and the mask images are set on the plurality ofimaging dates and times. In the example shown in FIG. 9, the same timezones on different dates are set as two imaging dates and times.

Next, the outline of the process performed by moving image outputcontroller 34 and the moving image displayed on moving image displaypart 48 of the monitoring screen will be described. FIGS. 10A and 10Bare explanatory diagrams showing a moving image on two imaging dates andtimes. FIGS. 11A and 11B are explanatory diagrams showing main parts ofmoving image display part 48 of the monitoring screen shown in FIG. 8.

FIG. 10A shows an imaged moving image on a first imaging date and time.FIG. 10B shows imaged moving image on a second imaging date and time.Moving image output controller 34 performs a process of generating amask image having an outline corresponding to an image area of a personin the imaged moving image on the first imaging date and time,generating a mask image having an outline corresponding to an image areaof a person in the imaged moving image on the second imaging date andtime, and superposing the mask image in a position corresponding to thebackground image generated by background image generator 33.Accordingly, as shown in FIG. 11, the mask image is displayed on movingimage display part 48 of the monitoring screen.

Herein, in the second exemplary embodiment, the mask images on differentimaging dates and times are displayed at the same time, and the maskimages on different imaging dates and times are superposed with eachother, in some cases. In the second exemplary embodiment, as a displaymethod of the mask images on different imaging dates and times, as shownin FIG. 11A, a normal display mode where the display is not performedbased on the order of priority can be selected, and as shown in FIG.11B, a priority display mode where the display is performed based on theorder of priority can be selected.

As shown in FIG. 11A, in the normal display mode where the display isnot performed based on the order of priority can be selected, thesuperposed part of the mask images on different dates and times aredisplayed to be discriminated from the part other than the mask images,that is, the parts not superposed with the mask images of the otherpersons. In the example shown in FIG. 11A, the superposed part of themask image is displayed with an image obtained by synthesizing each maskimage. For example, the part where a red mask image and a blue maskimage are superposed with each other is displayed in purple, and thesuperposed parts of the mask images are displayed at transmittanceobtained by adding transmittance of each mask image.

As shown in FIG. 11B, in the priority display mode in which the displayis performed based on the order of priority, the parts where the maskimages on different dates and times are superposed are displayed so thatthe mask image on the date and time having higher order of prioritycovers the mask image on the date and time having lower order ofpriority.

In the second exemplary embodiment as described above, since the maskimages by the plurality of imaging dates and times, for example, todayand yesterday are displayed, it is possible to simply grasp the imagingdate and time of the person appearing on the moving image, andtherefore, it is possible to efficiently perform the operation ofgrasping the action tendency of the person by the date and time. Herein,when the plurality of imaging dates and times are set as the same timepoints on different dates, it is possible to compare the actiontendencies of the persons for each date, and when the plurality ofimaging dates and times are set as the time zones in the morning and theevening of the same date, it is possible to compare the actiontendencies of the persons for each time zone in morning and evening.

Third Exemplary Embodiment

Next, a third exemplary embodiment will be described. The points notparticularly stated herein are the same as in the first exemplaryembodiment.

FIG. 12 is a functional block diagram showing a schematic configurationof PC 3 of a monitoring system according to the third exemplaryembodiment. In the third exemplary embodiment, in the same manner as inthe first exemplary embodiment, PC 3 includes monitoring unit 31, personimage analyzer 32, background image generator 33, moving image outputcontroller 34, and mask image setter 35, and PC 3 in the third exemplaryembodiment further includes background image analyzer 91.

Background image analyzer 91 performs a process of acquiring goodschanging information regarding appearance and disappearance of theproduct in the image area of the background in the imaged moving image.In this background image analysis process, by comparing the plurality ofbackground images obtained by removing the images of the persons(foreground images) from the imaged moving image, the changed states(appearance and disappearance) of the products are detected, and areainformation of the changed products (positional information regardingimage area of product) is acquired.

Moving image output controller 34 performs a process of generating andoutputting an output moving image displayed so as to discriminate thechanges (appearance and disappearance) of the products in the image areaof the background, based on the goods changing information output frombackground image analyzer 91. Herein, when the product disappears, theimage area where the product has been present in the moving image beforethe disappearance may be changed to a translucent mask image, and whenthe product appears, the image area of the product may be highlighted.

Various appropriate methods may be used for highlighting when theproduct appears. For example, the outline of the image area of theproduct may be displayed, or when the product is small and it isdifficult to recognize the product, an animation display such asblinking may be performed. When the product appears, the image area ofthe product may be changed to the translucent mask image, in the samemanner as when the product disappears. Appropriate mark images, forexample, images of signals such as O or X, may be displayed in theposition of the changed (appearance or disappearance) product.

In the third exemplary embodiment, the changed situations (appearanceand disappearance) of the products are displayed to be discriminated.Also, in the same manner as in the first exemplary embodiment, it ispossible to display the mask image of the person by attribute, maskimage setter 35 performs a process of setting a mask image according toan input operation by a user for selecting a mask image for eachattribute of the person, attribute determinator 38 performs a process ofdetermining attribute (gender, age, and the like) for each persondetected from the imaged moving image, and moving image outputcontroller 34 performs a process of generating and outputting an outputmoving image obtained by changing the image area of the person to a maskimage by attribute.

Next, a monitoring screen displayed on monitor 4 will be described. FIG.13 is an explanatory diagram showing the monitoring screen displayed onmonitor 4.

Even in the third exemplary embodiment, a monitoring screen having aconfiguration approximately similar to that of the first exemplaryembodiment (see FIG. 4) is displayed on monitor 4, and in the monitoringscreen, the mask image set by the attribute of the person and the maskimage showing the disappeared product are displayed on explanatory notedisplay part 47.

Next, the operation of setting the mask images by the attribute of theperson and the mask image of the changed states (appearance anddisappearance) of the product will be described. FIG. 14 is anexplanatory diagram showing a mask image setting screen displayed onmonitor 4.

This mask image setting screen sets the mask images by the attributes ofthe person and sets the mask images of the changed states (appearanceand disappearance) of the product. This mask image setting screenincludes person setting part 101, product setting part 102, and setcontents display part 103.

In the same manner as in the first exemplary embodiment (see FIG. 5),person setting part 101 includes person attribute selection part 104,mask image selection part 105, and setting button 106. When theattribute of the person (gender, age, and the like) is selected inperson attribute selection part 104, the mask image corresponding to theattribute of the person is selected in mask image selection part 105,and setting button 106 is operated, the set contents, that is, the maskimage by attributes, are displayed on set contents display part 103.

Product setting part 102 is provided with changed state selection part107, mask image selection part 108, and setting button 109. A userselects the changed states (appearance and disappearance) from apull-down menu in changed state selection part 107. A user selects themask image corresponding to the changed state selected in changed stateselection part 107 from a title menu in mask image selection part 108.When the changed state and the mask image corresponding thereto areselected in changed state selection part 107 and mask image selectionpart 108, and setting button 109 is operated, the set contents, that is,the mask image of the changed state of the product, are displayed in setcontents display part 103.

Next, the outline of the process performed by moving image outputcontroller 34 and the moving image displayed on moving image displaypart 48 of the monitoring screen shown in FIG. 13 will be described.FIGS. 15A, 15B, 16A, 16B, and 16C are explanatory diagrams showing mainparts of moving image display part 48 of the monitoring screen shown inFIG. 13.

In the third exemplary embodiment, in the same manner as in the firstexemplary embodiment (see FIG. 6A), the moving image imaged by camera 1is displayed as it is in moving image display part 48 in the normalplayback mode, and as shown in FIGS. 15A and 15B, the output movingimage obtained by changing the image area of the person to the maskimage is displayed in the mask playback mode after performing the maskimage setting.

Herein, as shown in FIG. 16A, the customer in front of the display shelfcarries the product away. As shown in FIG. 16B, when the productdisappears from the display shelf, the disappearance of the product isdetected by background image analyzer 91, and moving image outputcontroller 34 performs a process of changing the image area of thedisappeared product on the background image to the mask image.Accordingly, as shown in FIG. 16C, the mask image is displayed in aposition where the product has been present, in the mask playback mode.

The display of the mask image or the highlighting showing the changedstate (appearance and disappearance) of the product may be completedwhen the predetermined time has elapsed from the time point of thechange (appearance and disappearance) of the product. In the same manneras that of the mask image of the person, the mask image of the productis obtained by generating the outline of the image area of the productfrom the area information of the product (positional informationregarding image area of product) and covering the inside of the outlinewith a translucent image.

In the third exemplary embodiment as described above, since the maskimages are displayed by the attributes, it is possible to simply graspthe attribute of the person appearing on the moving image, andtherefore, it is possible to efficiently perform the operation ofgrasping the action tendency of the person for each attribute of theperson. Since the display is performed so as to discriminate the changeof the product (at least any one of appearance and disappearance), it ispossible for a user to discriminate the changing products, andtherefore, it is possible for a user to efficiently perform theoperation of grasping the changed state of the product with the actionof the person.

In the third exemplary embodiment, both the appearance and thedisappearance of the product is detected and the output image displayedso as to discriminate both the appearance and the disappearance isgenerated, but at least any one of the appearance and the disappearanceof the product may only be a target.

FIG. 17 is an explanatory diagram showing another example of themonitoring screen displayed on monitor 4. In the examples describedabove, panoramic moving images imaged by camera 1 which is theomnidirectional camera, are displayed on moving image display part 48,but in an example shown in FIG. 17, the moving images imaged by camera 1which is the omnidirectional camera, are displayed as they are.

Hereinabove, the invention has been described based on the specificembodiments, but the embodiments are merely examples, and the inventionis not limited to the embodiments. Not all of constituent elements ofthe monitoring apparatus, the monitoring system, and the monitoringmethod according to the invention shown in the embodiments arenecessary, and it is possible to appropriately remove and delete theconstituent elements in a range of not departing from the scope of theinvention.

For example, in the first to third exemplary embodiments, the example ofa retail store such as a convenience store has been described, but thereis no limitation to the retail store, and the invention can also beapplied to a store with an operation format other than that of theretail store. In addition, it is possible to set a monitoring area otherthan the store as a target.

In the first to third exemplary embodiments, as shown in FIG. 2, camera1 is set as the omnidirectional camera including an imaging range at 360degrees using a fisheye lens, but a camera including a predeterminedangle of view that is a so-called box camera can also be used.

In the first to third exemplary embodiments, the output moving imagewhere the mask images for each person are superposed on the backgroundimage is generated, but the mask image can only be displayed withoutdisplaying the background image.

In the first to third exemplary embodiments, the necessary settingmatters are input by selection from a pull-down menu or a title menu,but there is no limitation, and an appropriate input unit may beprovided on the screen, and the selection may be performed with a radiobutton or a numerical value may be directly input.

In the first to third exemplary embodiments, various configurationshaving respective characteristics are described, but theseconfigurations are not limited to the combination shown in each of thefirst to third exemplary embodiments, and the configurations shown ineach of the first to third exemplary embodiments can be appropriatelycombined with each other.

In the first to third exemplary embodiments, the process necessary forthe monitoring is performed by PC 3 provided in the store, but as shownin FIG. 1, the necessary process may be performed by PC 11 provided inthe head office, or cloud computer 21 configuring a cloud computingsystem. The necessary process may be shared by a plurality ofinformation processing apparatuses, and information may be transmittedto and received from the plurality of information processing apparatusesthrough a communication medium such as an IP network or a LAN. In thiscase, the monitoring system is configured with the plurality ofinformation processing apparatuses sharing the necessary process.

In the configurations described above, among the processes necessary forthe monitoring, at least a process having a large operation amount, forexample, the person detection process, may be performed by PC3 providedin the store. By configuring as described above, since the amount ofdata of necessary information in the remaining processes is small, evenwhen the remaining processes are caused to perform by an informationprocessing apparatus installed in another place different from thestore, for example, PC 11 installed in the head office, it is possibleto decrease a communication load, and therefore, it is easy to operatethe system by broad area network connection configuration.

Among the processes necessary for the monitoring, at least a processhaving a large operation amount, for example, the person detectionprocess, may be performed by cloud computer 21. By configuring asdescribed above, since an operation amount of the remaining process issmall, a high-speed information processing apparatus is not necessary onthe user side such as a store, and it is possible to reduce costshouldered by a user.

Cloud computer 21 may perform all of the necessary processes or at leastthe moving image output control process among the necessary processesmay be shared by the cloud computer 21. By configuring as describedabove, in addition to PC 3 and PC 11 provided in the store or the headoffice, it is also possible to display the moving image on a portableterminal such as smart phone 22 or tablet terminal 23, and therefore, itis possible to check the situations of the inside of the store in anarbitrary place such as places other than the store or the head office.

In the embodiment, PC 3 installed in the store performs the processnecessary for the monitoring, and the monitoring screen or the maskimage setting screen are displayed on monitor 4 of PC 3 to performnecessary input and output by PC 3, but the necessary input and outputmay be performed by an information processing apparatus other than theinformation processing apparatus which performs the process necessaryfor the monitoring, for example, PC 11 installed in the head office or aportable terminal such as tablet terminal 23.

Recently, a 4K television for achieving high image quality as a monitorhas been developed, and by employing a camera corresponding to such a 4Ktelevision, it is possible to improve discrimination performance of aperson. Even in an environment where there are a lot of persons, it ispossible for a user to simply grasp the action tendencies of the personfrom a mask playback moving image.

The monitoring apparatus, the monitoring system, and the monitoringmethod according to the invention have an effect of allowing a user toefficiently perform the operation of grasping the action tendencies ofthe person by the attribute of the person, and are useful as amonitoring apparatus, a monitoring system, and a monitoring method whichgenerate and output an output moving image obtained by changing an imagearea of a person detected from an imaged moving image of a monitoringarea to a mask image.

What is claimed is:
 1. A monitoring apparatus which generates andoutputs an output moving image obtained by changing an image area of aperson detected from an imaged moving image of a monitoring area to amask image, the apparatus comprising: a processor; and a memory whichstores instructions, wherein the monitoring apparatus further includes,as a configuration when the processor executes the instructions storedin the memory, a person image analyzer including a person detector whichdetects a person from the imaged moving image and acquires areainformation regarding each person and an attribute determinator whichdetermines attributes for each person detected from the imaged movingimage, a mask image setter which sets the mask image by the attribute,and a moving image output controller which generates and outputs theoutput moving image obtained by changing the image area for each personto the mask image by the attribute, based on the area information andthe attribute for each person output from the person image analyzer. 2.The monitoring apparatus of claim 1, wherein the mask image setter setsthe mask image by the attribute, according to an input operation by auser of selecting the mask image by the attribute.
 3. The monitoringapparatus of claim 1, wherein the mask image is set to discriminate theattribute, by changing at least any of display elements such as a color,shading, a pattern, and an outline, according to the attribute.
 4. Themonitoring apparatus of claim 1, further comprising: a background imagegenerator which generates a background image from the imaged movingimage, wherein the moving image output controller generates the outputmoving image where the translucent mask image is superposed on thebackground image.
 5. A monitoring system which generates and outputs anoutput moving image obtained by changing an image area of a persondetected from an imaged moving image of a monitoring area to a maskimage, the system comprising: cameras which image the monitoring area;and a plurality of information processing apparatuses, wherein any oneof the plurality of information processing apparatuses includes aprocessor, and a memory which stores instructions, and the monitoringsystem further includes, as a configuration when the processor executesthe instructions stored in the memory, a person image analyzer includinga person detector which detects a person from the imaged moving imageand acquires area information regarding each person and an attributedeterminator which determines attributes for each person detected fromthe imaged moving image, a mask image setter which sets the mask imageby the attribute, and a moving image output controller which generatesand outputs the output moving image obtained by changing the image areafor each person to the mask image by the attribute, based on the areainformation and the attribute for each person output from the personimage analyzer.
 6. A monitoring method which causes an informationprocessing apparatus to perform a process of generating and outputtingan output moving image obtained by changing an image area of a persondetected from an imaged moving image of a monitoring area to a maskimage, the method, when a processer of the information processingapparatus executes instructions stored in a memory, comprising: a stepincluding a step of detecting a person from the imaged moving image andacquiring area information regarding each person and a step ofdetermining attributes for each person detected from the imaged movingimage; a step of setting the mask image by the attribute; and a step ofgenerating and outputting the output moving image obtained by changingthe image area for each person to the mask image by the attribute, basedon the area information and the attribute for each person.
 7. Amonitoring apparatus which generates and outputs an output moving imageobtained by changing an image area of a person detected from an imagedmoving image of a monitoring area to a mask image, the apparatuscomprising: a processor; and a memory which stores instructions, whereinthe monitoring apparatus further includes, as a configuration when theprocessor executes the instructions stored in the memory, a date andtime setter which sets a plurality of imaging dates and times, accordingto an input operation by a user of designating the imaging date andtime; a person image analyzer including a person detector which detectsa person from the imaged moving image and acquires area informationregarding each person; a mask image setter which sets the mask image bythe imaging date and time; and a moving image output controller whichgenerates and outputs the output moving image obtained by changing theimage area of the person to the mask image by the imaging date and time,based on the area information regarding each person output from theperson image analyzer.
 8. The monitoring apparatus of claim 7, whereinthe person image analyzer includes an attribute determinator whichdetermines the attribute for each person detected from the imaged movingimage, and the mask image setter sets the mask image by the imaging dateand time and by the attribute, according to an input operation by a userof selecting the mask image by the imaging date and time and by theattribute.
 9. The monitoring apparatus of claim 7, wherein the movingimage output controller displays so as to discriminate a superposed partfrom a part other than the mask image, when the mask images of theplurality of persons are superposed with each other.
 10. The monitoringapparatus of claim 7, wherein the date and time setter applies the orderof priority to the plurality of dates and times, and the moving imageoutput controller displays a superposed part so that the mask image onthe imaging date and time having higher order of priority covers overthe mask image on the imaging date and time having lower order ofpriority, when the mask images of the plurality of persons aresuperposed with each other.
 11. The monitoring apparatus of claim 7,wherein the mask image is set to discriminate imaging date and time, bychanging at least any of display elements such as a color, shading, apattern, and an outline, according to the imaging date and time.
 12. Themonitoring apparatus of claim 7, further comprising: a background imagegenerator which generates a background image from the imaged movingimage, wherein the moving image output controller generates the outputmoving image where the translucent mask image is superposed on thebackground image.
 13. A monitoring system which generates and outputs anoutput moving image obtained by changing an image area of a persondetected from an imaged moving image of a monitoring area to a maskimage, the system comprising: cameras which image the monitoring area;and a plurality of information processing apparatuses, wherein any oneof the plurality of information processing apparatuses includes aprocessor, and a memory which stores instructions, and the monitoringsystem further includes, as a configuration when the processor executesthe instructions stored in the memory, a date and time setter which setsa plurality of imaging dates and times, according to an input operationby a user of designating the imaging date and time, a person imageanalyzer including a person detector which detects a person from theimaged moving image and acquires area information regarding each person,a mask image setter which sets the mask image by the imaging date andtime, and a moving image output controller which generates and outputsthe output moving image obtained by changing the image area of theperson to the mask image by the imaging date and time, based on the areainformation regarding each person output from the person image analyzer.14. A monitoring method which causes an information processing apparatusto perform a process of generating and outputting an output moving imageobtained by changing an image area of a person detected from an imagedmoving image of a monitoring area to a mask image, the method, when aprocesser of the image processing apparatus executes instructions storedin a memory, comprising: a step of setting a plurality of imaging datesand times, according to an input operation by a user of designating theimaging date and time; a step of detecting a person from the imagedmoving image and acquiring area information regarding each person; astep of setting the mask image by the imaging date and time; and a stepof generating and outputting the output moving image obtained bychanging the image area of the person to the mask image by the imagingdate and time, based on the area information regarding each person. 15.A monitoring apparatus which generates and outputs an output movingimage obtained by changing an image area of a person detected from animaged moving image of a monitoring area to a mask image, the apparatuscomprising: a processor; and a memory which stores instructions, whereinthe monitoring apparatus further includes, as a configuration when theprocessor executes the instructions stored in the memory, a person imageanalyzer including a person detector which detects a person from theimaged moving image and acquires area information regarding each personand an attribute determinator which determines attributes for eachperson detected from the imaged moving image, a background imageanalyzer which acquires goods changing information regarding a changedsituation of goods in an image area of a background in the imaged movingimage, a mask image setter which sets the mask image by the attribute,and a moving image output controller which generates and outputs theoutput moving image obtained by changing the image area for each personto the mask image by the attribute, based on the area information andthe attribute for each person output from the person image analyzer, anddisplayed so as to discriminate the changed situation of the goods inthe image area of the background, based on the goods changinginformation output from the background image analyzer.
 16. Themonitoring apparatus of claim 15, wherein the moving image outputcontroller displays at least any one of appearance and disappearance ofthe goods to be discriminated, as the changed situation of the goods.17. The monitoring apparatus of claim 15, wherein the mask image settersets the mask image by the attribute, according to an input operation bya user of selecting the mask image by the attribute.
 18. The monitoringapparatus of claim 15, wherein the mask image is set to discriminate theattribute, by changing at least any of display elements such as a color,shading, a pattern, and an outline, according to the attribute.
 19. Themonitoring apparatus of claim 15, wherein the moving image outputcontroller generates the output moving image obtained by changing theimage area of the changed goods to the translucent mask image, in orderto display the changed situation of the goods in the image area of thebackground to be discriminated.
 20. The monitoring apparatus of claim15, further comprising: a background image generator which generates abackground image from the imaged moving image, wherein the moving imageoutput controller generates the output moving image where thetranslucent mask image is superposed on the background image.
 21. Amonitoring system which generates and outputs an output moving imageobtained by changing an image area of a person detected from an imagedmoving image of a monitoring area to a mask image, the systemcomprising: cameras which image the monitoring area; and a plurality ofinformation processing apparatuses, wherein any one of the plurality ofinformation processing apparatuses includes a processor, and a memorywhich stores instructions, and the monitoring system further includes,as a configuration when the processor executes the instructions storedin the memory, a person image analyzer including a person detector whichdetects a person from the imaged moving image and acquires areainformation regarding each person and an attribute determinator whichdetermines attributes for each person detected from the imaged movingimage, a background image analyzer which acquires goods changinginformation regarding a changed situation of goods in an image area of abackground in the imaged moving image, a mask image setter which setsthe mask image by the attribute, and a moving image output controllerwhich generates and outputs the output moving image obtained by changingthe image area for each person to the mask image by the attribute, basedon the area information and the attribute for each person output fromthe person image analyzer, and displayed so as to discriminate thechanged situation of the goods in the image area of the background,based on the goods changing information output from the background imageanalyzer.
 22. A monitoring method which causes an information processingapparatus to perform a process of generating and outputting an outputmoving image obtained by changing an image area of a person detectedfrom an imaged moving image of a monitoring area to a mask image, themethod, when a processer of the image processing apparatus executesinstructions stored in a memory, comprising: a step including a step ofdetecting a person from the imaged moving image and acquiring areainformation regarding each person and a step of determining attributesfor each person detected from the imaged moving image; a step ofacquiring goods changing information regarding a changed situation ofgoods in an image area of a background in the imaged moving image; astep of setting the mask image by the attribute; and a step ofgenerating and outputting the output moving image obtained by changingthe image area for each person to the mask image by the attribute, basedon the area information and the attribute for each person, and displayedso as to discriminate the changed situation of the goods in the imagearea of the background based on the goods changing information.